[Performance] Allocate AlignedTVList primitive arrays lazily#18276
Conversation
There was a problem hiding this comment.
calculateRamSize() still multiplies the number of timestamp blocks by alignedTvListArrayMemCost(), which assumes that every column has a materialized primitive array in every block. That assumption is no longer valid after this change.
There was a problem hiding this comment.
Thanks, fixed in a335107. AlignedTVList now updates materializedValueArrayMemCost when getOrCreateValueArray() allocates a primitive array. calculateRamSize() uses the per-block cost without value primitive arrays plus the incrementally tracked materialized-array cost, so it no longer assumes dense value arrays or scans all blocks. The tracked state is also handled by clone, projection, column extension, and clear paths.
|
|
||
| if (acquireArray != 0) { | ||
| // memory of extending the TVList | ||
| memIncrements[0] += | ||
| acquireArray * alignedMemChunk.getWorkingTVList().alignedTvListArrayMemCost(); | ||
| for (TSDataType dataType : dataTypesInTVList) { | ||
| memIncrements[0] += acquireArray * AlignedTVList.valueListArrayMemCost(dataType); | ||
| } | ||
| } |
There was a problem hiding this comment.
This still charges the dense per-block cost for every existing column whenever a new block is acquired, while AlignedTVList.expandValues() now only adds null placeholders and allocates primitive arrays on their first actual write.
There was a problem hiding this comment.
Thanks, fixed in a335107. The aligned row, rows, and tablet estimators now split the charge into: (1) the base aligned-TVList block cost without value primitive arrays, (2) placeholder/bitmap cost for newly extended columns, and (3) primitive-array cost only when a successful non-null write first materializes that column/block. Sparse, null, and failed-write paths are covered by tests; the tablet path also skips per-row checks when both bitmap and results are absent.
|
Added regression coverage in commit Both tests intentionally fail on the current PR head: The second mismatch is 1056 bytes, exactly two unallocated INT64 primitive arrays (2 blocks x 528 bytes). The first shows that a sparse row and a dense row crossing the same block boundary are currently charged the same increment, although the dense row materializes one additional INT32 primitive array (272 bytes). Besides |
|


Description
Follow-up to #18264. This PR contains only the lazy primitive-array allocation changes on top of the merged dynamic-bitmap implementation.
nullplaceholders for newly added or unwritten aligned columns and allocate eachPrimitiveArrayonly on its first actual write.TsFileProcessormemory accounting for lazy primitive-array allocation.Performance
Compared master
91476b0df7with candidateb7f846ec98:Tests